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Testing  and  Assessment  in  a  Joint  Distributed  Environment 

Randy  Coonts  and  Thomas  M.  Fitzgerald,  Ph.D. 

United  States  Joint  Forces  Command,  Norfolk,  Virginia 

This  article  presents  an  overview  of  the  process  adopted  by  the  United  States  Joint  Forces 
Command  (USJFCOM)  and  its  partners  to  meet  requirements  to  conduct  testing  and 
assessments  in  a  joint  mission  environment.  The  process  is  based  on  a  distributed  joint  systems 
integration/interoperability  laboratory  concept.  The  methodology  describes  the  terms  of  the 
governance  to  ensure  joint  testing  and  assessments  meet  the  requirements  of  the  warfighter.  The 
process  is  defined  in  terms  of  joint  mission  threads  (JMTs),  associated  metrics ,  and  the  selection 
and  prioritization  of  JMTs.  This  is  followed  by  a  mapping  of  JMTs  into  available  exercises 
with  their  embedded  systems  and  a  risk  mitigation  analysis  to  ensure  feasibility.  Examples  of 
this  process  are  discussed  in  terms  of  exercise  data  collected  during  USJFCOM-supported 
exercises. 
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Today,  Joint  operational  systems  are 
allowed  to  “grow”  in  an  environment 
composed  of  agency  and  Service  devel¬ 
opment  and  testing  activities.  It  should 
come  as  no  surprise  that  many  of  these 
efforts  result  in  a  perfectly  adequate  Service  system  that 
does  not  perform  well  in  a  Joint  environment  where 
integration  and  interoperability  between  Services, 
agencies,  and  coalition  forces  are  important.  To  address 
this  shortfall,  for  command  and  control  systems,  the 
United  States  Joint  Forces  Command  (USJFCOM)  is 
developing  a  joint  systems  integration  and  interoper¬ 
ability  laboratory  (JSIIL)  concept.  The  laboratory  is 
“virtual”  and  composed  of  joint,  Service,  and  agency 
partners  and  their  accompanying  facilities  and  capa¬ 
bilities.  It  will  use  existing  venues,  networks,  and 
ranges  to  perform  joint  command  and  control  (C2) 
program  support.  The  title  of  the  laboratory  was 
chosen  to  reflect  the  goal  of  changing  the  way  C2 
information  technology-intensive  systems  are  devel¬ 
oped  and  fielded  and  to  make  the  whole  process  more 
agile  and  rapid. 

The  idea  (or  ideal)  of  testing  joint  programs  in  a 
joint  operational  environment  is  not  new.  In  fact  it  was 
influential  in  the  creation  of  the  Test  and  Resource 
Management  Center  in  2003  and  the  Testing  in  a  Joint 
Environment  Roadmap1  in  2004.  The  Director, 
Operational  Test  and  Evaluation  (DOT&E),  devel¬ 


oped  the  Testing  in  a  Joint  Environment  Roadmap, 
which  led  to  the  creation  of  the  Joint  Mission 
Environment  Test  Capability  (JMETC)  and  the  Joint 
Test  and  Evaluation  Methodology  (JTEM),  a  joint  test 
and  evaluation  (T&E)  project.  JMETC  provides  the 
infrastructure  for  joint  testing,  and  JTEM  provided  an 
initial  methodology  for  conducting  the  joint  tests.  Both 
of  these  are  critical  parts  of  the  capability  to  conduct 
joint  tests,  but  there  remained  at  least  one  missing 
piece:  the  joint  requirements  along  with  the  joint 
mission  threads  (JMTs)  to  provide  the  operational 
context  and  the  criteria  used  to  measure  effective 
performance. 

Background 

USJFCOM  began  evaluation  of  testing  in  a  joint 
environment  process  with  the  advent  of  responsibility 
for  joint  battle  management  for  C2  followed  by  joint 
capability  development  for  C2.  A  major  factor  in 
expanding  this  evaluation  was  the  coincidence  of 
USJFCOM  joint  capability  development  (USJFCOM 
J8)  providing  command  oversight  of  the  Coalition 
Warrior  Interoperability  Demonstration  (CWID)  pro¬ 
gram  and  the  evolution  of  test  threads  into  the  Empire 
Challenge  (EC)  and  Bold  Quest  events.  These  field 
demonstrations  and  assessments  have  been  annual 
events,  and  they  rely  on  support  from  a  number  of 
activities  including  USJFCOM  organizations,  specifi- 
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T&E  in  a  Joint  Environment 


cally  the  Joint  Systems  Integration  Center  (JSIC)  in 
Suffolk,  Virginia,  and  the  Joint  Fires  Integration  and 
Interoperability  Team  (JFIIT)  at  Eglin  Air  Force  Base, 
Florida.  Annually  these  organizations  identify  prom¬ 
ising  technologies  that  are  intended  to  address  current 
operational  concerns  in  the  command  and  control  and 
intelligence  processing,  interpretation,  and  dissemina¬ 
tion  areas.  However,  once  these  technologies  are 
identified,  the  long  process  leading  to  operational 
fielding  only  begins.  The  hypothesis  behind  the  JSIIL 
is  that  it  is  possible  to  accelerate  this  fielding  process  by 
making  more  effective  use  of  existing  venues  that 
support  not  only  demonstration  and  assessment  but 
also  development  testing,  operational  testing,  joint 
interoperability  testing  and  certification,  and  informa¬ 
tion  assurance  certification  and  accreditation. 

The  leadership  of  USJFCOM  J8  investigated 
requirements  for  testing  and  assessments  in  a  joint 
environment  and  determined  that  they  actively  partic¬ 
ipate  in  programs  to  be  tested  and  assessed,  potential 
venues  (e.g.,  CWID  and  EC),  “testing  and  assessment” 
activities  through  JFIIT  and  JSIC,  and  the  evolution  of 
test  threads  into  the  development  of  JMTs  (scenarios) 
in  support  of  command  and  control  requirements. 
What  was  still  needed  was  a  further  definition  of  the 
criteria  for  meeting  the  C2  requirements.  In  February 
2009  the  JSIIL  idea  was  proposed  to  the  Joint  Staff  J6, 
the  Joint  Interoperability  Test  Command  (JITC),  and 
the  Defense  Information  Systems  Agency  (DISA)  at  a 
session  to  determine  how  JITC  and  USJFCOM  could 
further  enhance  their  cooperative  efforts.  It  was 
unanimously  endorsed,  and  USJFCOM  set  out  to 
refine  and  develop  the  concept.  Eventually  the  effort 
resulted  in  a  Joint  C2  Network  Partnership  (JC2NP) 
Steering  Group. 

As  criteria  were  developed  (initially  called  net- 
enabled  Universal  Joint  Task  List  criteria  for  estab¬ 
lishing  better  key  performance  parameters  for  measur¬ 
ing  and  testing  C2)  it  became  evident  that  this 
approach  would  need  to  engage  other  partners  to 
include  the  joint  testing  and  the  DOT&E  and  Service 
operational  test  communities.  So  the  dialogue  with  the 
DOT&E  and  Service  test  activities  was  initiated. 
Again  there  was  widespread,  albeit  qualified,  accep¬ 
tance  of  the  concept  provided  it  did  not  increase  cost 
and  resulted  in  more  rapid  fielding.  The  focus  was  on 
the  development  of  a  Strategic  National  (SN)  Univer¬ 
sal  Joint  Task  (UJT)  for  interoperability  called  SN  7.8 
UJT. 

In  March  2009,  USJFCOM  began  the  development 
of  the  JSIIL  concept  with  parallel  actions  to: 

•  develop  an  agreed  upon  set  of  JMTs  that  would 
provide  operational  context  for  measuring  im¬ 


provements  for  C2  capabilities  in  a  full  range  of 
Joint  operational  requirements; 

•  create  a  set  of  SN  7.8  UJTs-based  criteria  that 
would  provide  a  means  of  measuring  the 
attainment  of  the  mission  thread  capabilities; 

•  begin  the  process  of  matching  C2  programs  (both 
legacy  and  new)  to  a  mission  thread  and  a  set  of 
SN  7.8  UJT  criteria; 

•  align  the  evaluation  of  programs,  by  mission 
thread  and  SN  7.8  UJT,  to  an  existing  venue,  e.g., 
CWID  and  EC,  to  begin  investigating  the 
concept;  and 

•  define,  coordinate,  and  establish  a  governance 
forum  for  the  testing  and  certification  process  to 
provide  support  and  oversight  of  the  effort. 

Since  March  2009,  USJFCOM  has  concentrated  on 
the  development  of  JMTs  and  an  SN  7.8  UJT.  In  a 
novel  approach  the  team  built  and  applied  the  SN  7.8 
UJT  in  an  existing  venue,  the  Joint  Expeditionary 
Force  Experiment.  This  first  addition  of  a  Service 
assessment  venue,  in  this  case  Air  Force,  helped  define 
other  venues  that  would  be  needed  to  make  the  process 
persistent  throughout  the  year  rather  than  just 
episodic.  Two  lessons  were  learned  in  this  first  effort: 
additional  definition  of  the  SN  7.8  UJTs  was  needed, 
and  data  to  support  a  rigorous  objective  measurement 
process  were  not  routinely  captured  in  the  demonstra¬ 
tion  and  assessment  events.  Both  of  these  areas  are 
being  further  developed  in  preparation  for  future 
events. 

Process 

USJFCOM,  in  cooperation  with  DISA/JITC,  the 
joint  staff,  the  DOT&E  community,  and  the  Service 
operational  test  agencies  (OTAs),  has  just  started  on  a 
journey  to  significantly  enable  the  joint  testing  and 
assessment  process.  There  has  been  widespread  en¬ 
dorsement  of  the  need  for  improvement  and  the 
deliberate  approach  that  is  being  followed  by  USJF¬ 
COM.  The  goal  is  to  provide  an  environment  that  will 
support  joint  testing,  assessments,  and  certification 
through  a  “one  team,  one  time,  one  set  of  conditions” 
approach.  A  lot  of  hard  and  tedious  work  lies  ahead, 
but  with  continued  focus  on  the  goal  to  rapidly  field 
needed  operational  capability,  the  effort  will  succeed. 
The  joint  testing/assessment  process  up  to  the 
development  of  JMTs  is  summarized  in  Figure  1. 

From  a  USJFCOM  perspective,  the  governance  for 
joint  testing  comes  from  USJFCOM’s  Unified  Com¬ 
mand  Plan,2  which  includes  the  following: 

a.  Providing  recommendations  to  ensure  integration 
of  Service,  defense  agency,  interagency,  and 
multinational  capabilities  development; 
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Joint  Mission  Thread 


Forces,  Threats,  L,V,C 
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(Conditions,  Circumstances) 


JointTest  Environment 
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Ensure  integration 
and  interoperability 
(UCP,  TIJE  Roadmap) 
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Governance 


JMT priorities  set  by 
COCOMS(IPLs, 
JUONs),  JCOA, 
Services,  Agencies, 
JROC,  JSIIL 


Figure  7.  Joint  testing/assessment  process. 


b.  in  coordination  with  the  chairman,  leading  the 
development  of  joint  concepts,  requirements,  and 
integrated  architectures  for  joint  C2  to  ensure 
integration  and  interoperability  from  the  tactical 
level  through  the  interface  with  the  global  level; 

c.  in  coordination  with  the  chairman,  supporting 
the  development  of  fully  interoperable  joint 
warfighting  capabilities  and  concepts. 

In  addition  USJFCOM  is  a  full  partner  with 
DOT&E,  which  leads  the  development  and  execution 
of  the  Testing  in  a  Joint  Environment  Roadmap.  The 
Roadmap  establishes  the  policy  that  developing  and 
fielding  joint  force  capabilities  require  adequate, 
realistic  T&E  in  a  joint  operational  context. 

DOT&E  also  leads  the  Testing  in  a  Joint  Environ¬ 
ment  Roadmap  Senior  Steering  Group  (TSSG)3  and 
has  chartered  a  TSSG  Advisory  Group  (TSSG-AG)  to 
investigate  the  ramifications  of  the  Roadmap.  USJF¬ 
COM  is  a  member  of  each  group.  The  TSSG-AG  uses 
the  following  working  definitions  for  the  elements 
shown  in  Figure  1\ 

•  joint  operational  environment:  The  environment 
of  land,  sea,  and/or  airspace  within  which  a  joint 
force  commander  employs  capabilities  to  execute 
assigned  missions; 

•  Joint  Publication  (JP)  1-02:  4  A  composite  of 
the  conditions,  circumstances,  and  influences 
that  affect  the  employment  of  capabilities  and 
bear  on  the  decisions  of  the  commander; 

‘  JP  3-05expands  on  the  JP  1-02  definition:  it 
encompasses  physical  areas  and  factors  (of  the 
air,  land,  maritime,  and  space  domains)  and 


the  information  environment,  which  include 
the  adversary,  friendly,  and  neutral  systems 
that  are  relevant  to  a  specific  joint  operation; 

•  joint  mission  environment:  “A  subset  of  the  joint 
operational  environment  composed  of  force  and 
non-force  entities;  conditions,  circumstances  and 
influences  within  which  forces  employ  capabili¬ 
ties  to  execute  joint  tasks  to  meet  a  specific 
mission  objective”; 

•  joint  test  environment  (TSSG-endorsed  defini¬ 
tion):  The  appropriate  combination  of  represen¬ 
tative  systems,  forces,  threats,  and  environmental 
conditions  assembled  for  test  in  a  joint  mission 
environment  to  support  evaluations.  These  rep¬ 
resentations  can  be  live,  virtual,  and  constructive, 
or  distributed  combinations  thereof.  The  joint 
test  environment  should  also  include  certified 
models  and  simulations,  test  and  evaluation 
master  plans  from  the  program  managers  and 
Program  Executive  Offices; 

•  joint  mission  thread:  An  operational  and  technical 
description  of  the  end-to-end  set  of  activities  and 
systems  that  accomplish  the  execution  of  a  joint 
mission  (Ref.  Chairman  Joint  Chiefs  of  Staff 
Instruction  [CJCSI  6212. 01E,  Dec  08]).  This 
definition  is  currently  being  revised  by  the  joint  staff. 

For  the  past  few  years  USJFCOM  has  been  using  an 
expanded  view  of  the  joint  mission  environment  as  a 
multiservice  operation  described  in  the  context  of  a 
joint  integrated  architecture.  Also  under  consideration 
is  including  the  role  of  a  joint  task  force  commander 
and  coalition  forces  in  the  definition. 
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JMT  generic  data  description- totally 
reusable  architecture-based  info  sets 


Tierl: 

JMT:  AV-1(High  Level),  AV-2,  OV-1,  OV-2,  OV4,  OV-5b, 

SV-1,  High  Level  Measures,  HighLevel  Executable  Architecture  (BPMN) 


Tier  2: 

JMT  "Strands"  (Additional)  AV-1  (Specific),  AV-2,  OV-2,  OV-3,  OV4,  OV-5b,  OV-6c, 
SV-1,  SV-3,  SV-5a,  SV-6,  SV-IOc,  SvcV-3a,  SvcV-5,  Scenario  Based  Executable 
Architecture,  or  documents  that  show: 

•  Node/System  Pairing 

•  Message  Order 

•  Distributions 

•  Timings 

•  Decision  processes 

•  System  attributes 

•  System  Functionality  (IAWJCSFL) 

•  Information  Exchange  Requirements  (lERs) 

•  Message  composition 

•  Interoperability  Matrix 

•  Data  Exchange  Requirements  (DERs) 

•  System  Capabilities 


Tool  suite  will  include  JACAE,  ArchitectureDriven  Analysis  (ADA)  EAs,  augmented  by: 

•  JDEIS 

•  JCSFL 

•  Joint  Style  Guide  (ArmysJointTest  Threads  +) 


JMT  “Strand"  Tier  2  information 
represents  specific  documentation 
required  to  answer  a  particular  question  or 
solve  a  problem.  JMT  Strands  are  unique 
JMT  segments  that  will  have  specific 
actors  for  each  node,  decomposed 
measures,  might  use  a  Servicespecific 
set  of  TTPs  or  CONOPS,  may  be  AOR 
specific  or  use  a  unique  set  of  systems 
and  apps  -  all  subset  of  Tier  1  information 
(OV-5,  SV-1,  etc.). _ 


Figure  2.  JMT  development  products. 


Over  the  past  several  years  there  has  been  consid¬ 
erable  interest  in  understanding  and  executing  joint 
military  operations.  One  approach  to  this  understand¬ 
ing  has  been  the  development  and  use  of  Department 
of  Defense  (DoD)  Architecture  Framework6  products. 
Combatant  commands  (COCOMs),  Services,  and 
agencies  have  spent  considerable  time  and  energy 
developing  these  products  to  understand  and  manage 
the  complexity  associated  with  military  operations. 
USJFCOM  in  particular  has  worked  closely  with  the 
services  and  agencies  to  develop  architecture  templates 
describing  major  combat  operations,  homeland  securi¬ 
ty,  and  disaster  relief,  among  other  areas.  These 
products  have  been  used  to  support  joint  task  force 
(JTF)  organizational  planning  for  the  COCOMs, 
demonstrations,  exercises,  and  test  planning.  They 
have  also  been  coordinated  with  the  Master  Training 
Guide  for  the  training  of  JTFs. 

One  of  the  major  mission  areas  of  USJFCOM  as 
assigned  through  the  under  secretary  of  Defense  is  as 
co-lead  of  the  Command  and  Control  Capability 
Portfolio  manager,  whose  task  is  to  identify  opportu¬ 
nities  to  improve  joint  interoperability,  eliminate 
redundancies,  identify  gaps,  and  streamline  the  acqui¬ 
sition  and  budget  processes  to  support  the  C2  needs  of 
the  Joint  force  commander.  Explicit  to  this  responsi¬ 
bility  is  the  mandate  to  support  integrated  architectures 
for  the  portfolio. 

During  this  time  frame  there  has  also  developed  a 
need  to  provide  a  joint  environment  to  support  testing 


and  assessment  of  the  increasing  numbers  of  joint 
systems  being  introduced  into  the  acquisition  process. 
DOT&E  has  recognized  this  requirement  and  has 
published  the  DOT&E  Roadmap,  established  JTEM, 
and  created  JMETC,  which  focuses  on  the  required 
persistent  infrastructure  to  conduct  joint  tests.  The 
joint  mission  environment,  however,  remains  to  be 
defined  from  the  operational  perspective  for  a  Joint 
program  manager  who  has  to  test  systems  or  systems  of 
systems  in  the  joint  mission  environment. 

Architectures  and  joint  mission  threads 

The  architectural  efforts  at  USJFCOM,  coordinated 
with  the  services,  extend  from  the  operational  JTF 
headquarters  to  the  force  component  commanders 
down  to  the  tactical  level  and  at  least  one  level  up  to 
the  strategic  level.  It  includes  everything  from  mapping 
of  joint  capability  areas  to  Universal  Joint  Task  Lists  to 
organizational  activities  to  Joint  Common  System 
Function  Lists  to  systems.  It  is  within  this  architectural 
context  that  the  joint  mission  thread  can  be  developed 
with  associated  metrics  and  measures  that  support 
testing  and  training,  which  can  be  used  to  identify 
gaps,  redundancies,  and  potential  solutions. 

The  approach  currently  used  at  USJFCOM  is 
similar  to  the  approach  used  in  the  earlier  joint  battle 
management  for  C2  effort  at  USJFCOM,  where  a 
JMT,  e.g.,  Joint  Close  Air  Support,  was  assessed 
analytically  and  assessed/tested  in  the  context  of  a  joint 
integrated  architecture.  That  approach  can  be  de- 
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Initial  JMT  List 

•  Counter  IED 

•  Joint  Close  Air  Support  (JCAS) 

•  Air  and  Missile  Defense 

•  Joint  Personnel  Recovery  (JPR) 

•  CND/CNA/CNE 

•  EW/EA 
•CWMD 

•  Global  Force  Management 

•  Medical  Evacuation 

•  Joint  Suppression  of  Air  Enemy 
Defense  (JSEAD) 

•  Joint  Fires 

•  Time  Sensitive  Targeting 

•  Anti  Air  Warfare 

•  Interdiction 

•  Counter  Mine 

•  Mine  Operations 

•  Counter  Drug 

•  Amphibious  Assault 
•Tactical  Airlift 

•  PSYOP 

•  Maritime  Interception 

•  Defensive  Countermeasures 

•  Military  Deception 

•  Non  Combat  Evacuation 

•  Non-Lethal  Attack 

•  Interagency  Interoperability 

•  HA/DR 

•  Integrated  Tactical  Warning  &  AA 

•  Strategic  Attack 

•  DSCA 


Joint  Mission  Thread  (JMT):  an  operational  and  technical 
description  of  the  end-to-end  set  of  activities  and  systems 
that  accomplish  the  execution  of  a  joint  mission.  (CJCSI 
62 12. 01 E) 

Joint  Close  Air  Support 


Mission 
Event  No. 

Description 

1 

Unit  detects  target 

2 

Commander  decides  to  request  CAS 

3 

Unit  notified  TACP 

4 

TACP  passes  request  to  ASOC 

6 

ASOC  coordinates  with  senior 
ground  HQs  which  epprove  request 

6 

ASOC  assigns  on-call  aircraft 

7 

CRC  send  aircraft  to  contact  point 
(CP) 

8 

AWACS  passes  critical  updates  to 
aircraft 

9 

JTAC  briefs  aircraft 

10 

Aircraft  depart  Initial  point  (IP) 

11 

JTAC  controls  CAS  aircraft 

12 

Bombs  on  target 

13 

Assessment 

Joint  Context  to  Clarify  Joint  Requirements  & 
Improve  Interoperability  and  Integration 


Figure  3.  Proposed  JMT  initial  list. 


scribed  as  architecture-driven  analysis;  the  approach 
recommended  here  for  the  T&E  community  is 
architecture-driven  testing  and  assessment.  The  com¬ 
mon  theme  of  an  integrated  architecture  provides 
continuity  and  unity  of  effort  across  the  DoD.  Some 
may  prefer  to  call  this  a  JMT  approach  to  testing  and 
assessment  vice  an  architecture  approach.  That  is 
acceptable  because  JMTs  are  to  a  large  part  made  up 
of  architectural  products.  This  is  a  natural  evolution  of 
the  communities  of  interest  to  understand  and  deal 
with  the  complexity  of  testing  and  assessing  joint  C2 
capabilities.  Figure  2  illustrates  the  JMT  development 
products. 

As  can  be  seen  in  Figure  2,  many  of  the  JMT 
products  are  DoD  Architecture  Framework  products 
composed  of  operational  views,  system  views,  and  all 
views  and  may  include  tactics,  techniques,  and 
procedures  for  particular  JMTs.  Tool  suites  can  also 
be  included,  e.g.,  Joint  C2  Architecture  and  Capability 
Assessment  Enterprise,  a  USJFCOM  architecture 
development  and  data  base  tool  augmented  by  the 
Joint  Doctrine  Education  and  Training  Information 
System  and  the  Joint  Common  System  Function  List 
(JCSFL).  The  JMTs  are  broken  down  into  higher 


resolution  levels  called  tier  levels.  Through  efforts  of 
the  Joint  Mission  Thread  Architecture  for  Testing 
Working  Group  (JMTAT  WG),7  composed  of  several 
hundred  participants  from  COCOM,  Services,  and 
agencies,  this  community  of  interest  is  converging  on  a 
workable  (if  not  yet  common)  definition  of  a  JMT. 
From  the  acquisition  and  tester  perspectives,  JMTs  can 
provide  the  common  solution  to  the  needs  of  both 
communities  and  are  being  widely  recognized  as  such. 

The  proposed  JMT  initial  list  under  consideration 
for  investigation  by  USJFCOM  in  the  future  is  shown 
in  Figure  3.  Also  depicted  is  a  high-level  nodal 
diagram  for  a  Joint  Close  Air  Support  mission.  If 
USJFCOM  and  the  testing  community  are  to  speed  up 
the  delivery  of  capability  to  the  warfighter,  there  needs 
to  be  multiple  partnerships  in  a  parallel  development 
and  assessment  of  these  JMTs.  Currently  a  JMT 
Concept  of  Operations  document  is  in  the  Joint  Staff 
Action  Processing  review  external  to  USJFCOM. 

Up  to  this  point  the  process  has  been  one  of  analysis, 
sometime  referred  to  as  a  desktop  analysis  phase, 
requiring  constructing  JMTs  at  different  levels  of 
fidelity,  developing  the  associated  metrics,  developing 
executable  architecture,  and  running  appropriate  mod- 


388  ITEA  Journal 


T&E  in  a  Joint  Environment 


Activities 
OVs  &  SVs 
Metrics(KPPs) 


Joint  Missioi 


nues 


Exercises 

JCTDs 

CWID 


Joint 

Service 

Coalition 


Capabilities 

Gaps 

Shortfalls 

Problems 


Figure  4.  Selective  process. 


els  and  simulations  to  provide  guidance  on  the  possible 
way  ahead  to  a  more  formal  testing  and  assessment 
process. 

The  selection  process  to  move  toward  a  testing  and 
assessment  approach  of  C2  capabilities  through  JMTs 
is  shown  in  Figure  4,  where,  along  with  the  JMT  of 
interest,  suitable  venues  and  necessary  systems  that  are 
available  need  to  be  considered.  Venues  can  include 
training  exercises,  CWID,  joint  capability  technology 
demonstrations  (JCTDs),  multiservice  events,  etc. 
Systems  can  include  Service,  joint,  and  coalition 
systems  in  various  stages  of  development  from  concept 
to  development  to  programs  of  record  to  postdeploy¬ 
ment.  The  intersection  of  the  three  Venn  diagrams  in 
Figure  4  represents  a  synchronized  compilation  of 
threads,  systems,  and  venues  to  be  tested/assessed. 
Once  developed  and  tested,  these  JMTs  are  reusable 
and  provide  an  efficiency  in  follow-on  tests  that  may 
need  pieces  of  these  JMTs  to  investigate  and  test 
related  JMTs. 

There  is  still  a  requirement  for  a  set  of  selection 
criteria  in  choosing  an  appropriate  subset  of  threads, 
systems,  and  venues. 

USJFCOM  has  partnered  with  DOT&E  to  partic¬ 
ipate  in  its  Interoperability  and  Information  Assurance 
program.  Under  a  congressional  mandate  administered 
by  DOT&E,  an  Interoperability  and  Information 
Assurance  assessment  must  be  conducted  in  at  least 
one  COCOM  exercise  per  year  per  COCOM. 
USJFCOM  has  been  working  with  DOT&E  in 
developing  JMTs  to  support  several  COCOM  exer¬ 


cises.  DOT&E  provides  data  collectors  primarily 
through  the  OTAs  for  these  exercises.  Because  of 
mutual  support,  these  exercises  are  of  high  interest  to 
USJFCOM  as  likely  venues  to  pursue. 

The  Training  Directorate  of  USJFCOM,  J7, 
conducts  several  major  training  exercises  per  year 
including  mission  rehearsal  exercises.  While  training 
exercise  objectives  are  of  primary  concern,  there  is  a 
formal  mechanism  for  introducing  technology  ele¬ 
ments,  including  operational  test  events,  into  a  training 
exercise  without  compromising  the  training  objectives. 
The  mechanism  is  a  slightly  outdated  acronym,  JETA, 
standing  for  Joint  Experimentation,  Joint  Test  & 
Evaluations,  and  Advanced  Concept  &  Technology 
Demonstrations.  It  is  considered  slightly  outdated 
because  Advanced  Concept  &  Technology  Demon¬ 
strations  have  been  replaced  with  JCTDs.  With  the 
shortage  of  troops  available  for  many  venues,  it  is  a 
matter  of  necessity  and  efficiency  that  the  common 
goals  of  the  testing  and  training  communities  be 
aligned  wherever  possible.  Within  USJFCOM  and 
with  DOT&E  there  is  an  extensive  effort  to  take 
advantage  of  the  widely  perceived  efficiencies  to  be 
gained  by  both  the  testing  and  training  communities 
through  closer  cooperation.  A  simple  example  is  the 
certification  by  the  training  community  of  a  particular 
tool  used  to  collect  data  during  a  test.  Following 
certification  by  the  training  community,  there  was  no 
problem  using  that  tool  to  collect  valuable  test  data  in  a 
training  exercise.  Examples  like  this  are  important  so 
that  extensive  test  tools  being  developed  by  InterTEC8 
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Figure  5.  Integration  of  test  and  training  networks. 


can  be  used  in  training  exercises  without  fear  of 
jeopardizing  training  objectives.  A  more  complicated 
example  relates  to  the  gains  to  be  had  when  taking 
simultaneous  advantage  of  test  and  training  ranges, 
another  area  of  interest  to  both  USJFCOM  and 
DOT&E. 

USJFCOM  is  also  responsible  for  conducting  the 
CWID  program,  which  provides  numerous  opportu¬ 
nities  for  the  United  States  to  investigate,  assess,  and 
test  new  technologies  with  coalition  partners.  USJF¬ 
COM  is  also  a  major  participant  in  the  JCTD 
program,  which  is  another  opportunity  to  provide 
demonstrated,  operationally  assessed,  and  tested  tech¬ 
nologies  for  meeting  warfighter  requirements.  There 
are  also  numerous  multiservice  events  that  provide 
additional  opportunities  for  testing  and  assessment. 

In  selecting  from  various  events,  a  simple  risk 
mitigation  approach  is  used  that  considers  time, 
funding,  program  synchronization,  measurable  metrics, 
certified  tools,  verified  models  and  simulations,  and 
suitable  facilities.  At  this  point  a  recommendation  of 
where  and  how  to  conduct  the  test/assessment  is  made. 

Distributed  testing/assessment 

Figure  5,  compiled  by  the  JMETC  office,  shows 
integration  of  test  and  training  networks/sites  that 
could  be  used  to  conduct  a  specific  test/assessment. 


Approximately  40  of  these  sites  are  linked  together  via 
the  efforts  of  JMETC  with  several  more  sites  to  be 
added  over  time,  providing  a  sustainable  base  infra¬ 
structure  to  support  testing/assessment  and  training. 
Based  on  the  analysis  to  identify  an  appropriate  set  of 
threads,  systems,  and  venues  under  the  JSIIL  concept, 
a  suitable  set  of  facilities  can  be  selected  for  the  test/ 
assessment  process.  The  Joint  Systems  Integration 
Center  and  the  JFIIT  are  part  of  the  USJFCOM  J8 
Directorate  and  would  be  considered  like  any  other 
organization  as  a  possible  candidate  to  participate  in 
the  test/assessment  process.  Neither  JSIC  nor  JFIIT  is 
a  formal  test  agency,  although  both  have  worked  with 
the  OTAs  and  JITC  by  providing  supporting  assess¬ 
ments.  The  ultimate  selection  is  a  consensus  provided 
by  the  JC2NP  Steering  Group.  An  early  example  of 
this  selection  approach  for  a  distributed  event  was 
evaluated  with  the  Simulation  and  Analysis  Facility 
(Menke  and  March  2009)  at  Wright-Patterson  Air 
Force  Base,  Ohio,  in  support  of  the  Air  Ground 
Integrated  Layer  Exploration  project,  which  provided 
valuable  lessons  learned.  As  additional  sites  are  needed 
they  become  part  of  the  configuration. 

A  draft  JSIIL  organization  chart  is  shown  in 
Figure  6.  JSIIL  would  serve  as  a  focal  point  for  joint 
integration/interoperability  assessments.  It  would  pro¬ 
vide  a  coordinating  and  facilitation  service;  harness  the 


390  ITEA  Journal 


T&E  in  a  Joint  Environment 


JC2NP  SG 


JSIIL  Director 


JSIIL  Coordination  Cell 
(JCC) 


J8/JCD 

(Divisions 

JSIC 

JFIIT) 


J7/JWFC 


JC2NP  0-6  WG 

I 


JS  IILWG 


DRAFT 


Figure  6.  JSIIL  organization. 


conditions  present  in  operationally  realistic  joint, 
coalition,  and  combined  arms  environments;  provide 
alignment  of  JMT  development  and  supporting 
systems  engineering  efforts;  and  provide  objective 
feedback  and  recommendations  to  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology,  and  Logistics; 
DOT&E;  Services;  and  the  TSSG  for  determining 
annual  priorities  and  establishing  a  longer  term  plan  of 
action  and  milestones  for  JSIIL. 

The  JSIIL  director  is  provided  by  USJFCOM  J8  and 
coordinates  findings  of  the  0-6  Working  Group.  The 
director  briefs  those  findings  to  the  JC2NP  Steering 
Group.  The  JC2NP  Steering  Group  is  the  senior 
governance  body  with  a  tri-chair  under  consideration 
that  approves  a  JSIIL  priority  list  and  the  JSIIL  annual 
project  plan.  The  JSIIL  Coordination  Cell  is  chaired 
by  USJFCOM’s  Joint  Capability  T&E  division  head. 
JSIC  and  JFIIT  from  USJFCOM  J8,  the  USJFCOM 
J7  Joint  Training/Joint  Warfighting  Center  Director¬ 
ate,  and  the  J9  Joint  Concept  Development  and 
Experimentation  Directorate  support  the  JSIIL  Work¬ 
ing  Group. 

In  summary,  the  JSIIL  joint  assessment  process  is 
shown  in  Figure  7.  It  is  a  virtual  concept  of  a 
consortium/partnership  of  organizations  brought  to¬ 
gether  to  satisfy  the  requirement  to  assess/ test  in  a  joint 
environment.  It  is  made  up  of  a  variety  of  organiza¬ 
tions,  e.g.,  DDR&E;  DOT&E;  the  Under  Secretary  of 
Defense  for  Acquisition,  Technology,  and  Logistics; 
the  chairman;  the  Joint  Chiefs  of  Staff;  the  Under 


Secretary  of  Defense  for  Personnel  and  Readiness;  the 
Assistant  Secretary  of  Defense  for  Networks  and 
Information  Integration/Department  of  Defense  chief 
information  officer;  the  Under  Secretary  for  Intelli¬ 
gence;  OTAs;  Test  and  Resource  Management  Center 
ranges;  JMETC;  training  ranges;  DISA;  JITC;  Service 
Warfare  Centers;  academia;  industry;  agencies;  and 
COCOMs.  It  is  in  effect  an  effort  to  marshal  this 
nation’s  technological  expertise  to  speed  capabilities 
into  the  hands  of  the  warfighter  through  more  efficient 
testing  in  the  joint  environment.  This  is  not  a  task  that 
can  be  done  by  any  one  organization. 

In  2007,  an  article  appeared  in  ITEA  Journal  entitled 
“‘Born-Joint’ — Is  the  T&E  Community  Ready?” 
(Hutchison  2007).  The  answer  then  was  no,  not 
without  additional  efforts  in  the  areas  of  policy, 
funding  for  JMETC,  completion  of  JTEM,  and 
creation  of  joint  environments  for  training  and  T&E 
and  leadership.  With  advances  in  all  these  areas  and 
the  subsequent  development  of  more  refined  joint 
architectures  and  JMTs,  creation  of  aJC2NP,  support 
of  a  JSIIL  concept,  and  a  more  open  dialogue  among 
the  communities  of  interest,  considerable  progress  has 
been  made.  A  JSIIL  endorsement  letter  by  flag  officers, 
general  officers,  and  Senior  Executive  Service  members 
from  the  testing  community;  acquisition  community 
joint  staff;  and  USJFCOM  has  recently  been  signed. 
This  endorsement,  coupled  with  an  expansion  of  the 
existing  partnerships,  strengthens  the  foundation  for 
continued  progress  in  joint  distributed  testing  and 
assessment.  □ 


Randy  Coonts  manages  the  Joint  Capabilities  Test  and 
Evaluation  Division  supporting  the  director,  Joint 
Capability  Development  on  behalf  of  the  commander, 
Joint  Forces  Command.  He  supports  JFCOM  oversight  of 
all  Department  of  Defense  (DoD)  joint  command  and 
control  programs.  He  serves  as  an  action  officer  for 
JFCOM  to  a  wide  variety  of  DoD  agencies  and 
multinational  and  industrial  forums.  He  works  with  the 
Uniformed  Services  and  Combatant  Commands,  focusing 
on  the  coordination  of  DoD  test  and  evaluation,  capability 
develop?nent,  resourcing,  and  acquisition  processes  in 
support  of  warfighter  requirements.  He  is  currently  leading 
the  newly  formed  Joint  Integration  Interoperability 
(JSIIL)  Coordination  Cell. 

Dr.  Thomas  M.  Fitzgerald  received  his  doctor  of 
philosophy  degree  in  physics  from  Brown  University.  He 
has  been  a  research  scientist  for  NASA;  RAND  Corpora¬ 
tion,  in  Santa  Monica,  California;  and  Navy  R&D 
laboratories  including  the  Naval  Underwater  Systems 
Center,  Newport,  Rhode  Island,  and  the  Space  and  Naval 


31(3)  •  September  2010  391 


Coonts  &c  Fitzgerald 


COCOM  Exercise  /  Empire 
Challenge  /  CWID  /  Bold  Quest  / 
AGILE/JEFX 


JSIIL  Scope 

•  Responsive  to 
Warfighters  and 
Program  Managers 

•  Assessment  in  support 
of  experimentation, 
training,  integrated 
testing 

•  Cross-portfolio:  C2, 
Net-Centric,  BA, 

Logistics 

•  Consortium  of 
Services,  Combatant 
Commands,  Agencies  to 
form  an  operationally 
realistic,  persistent 
environment  for  Joint 
C2  analysis  and 
assessment 


r  Mission 
Partners: 
DOT&E,  AT&L, 
DISAJITC, 
TRMC 


Joint 

Systems 

integration 

Center 


JSIIL 


Acquisition, 
Testing,  Training  & 
Experimentation 
Communities 


JSIIL  Partnership 


Industry  Lab’ 


The  How 

Create  "a  joint  scenario  mission  threat  or  joint 
context  that  the  services  can  use  to  test  their 
equipment.  .  .  in  order  to  ensure  that  it  will 
meet  a  joint  need  to  satisfy  a  joint 
requirement. " 


Air  Force 


Joint  Fires 
integration  & 
Interoperability 
Team 


Figure  7.  Joint  assessment. 


Warfare  Center ,  San  Diego,  California.  Specialties  include 
microwave  and  undersea  acoustics,  modeling  and  simula¬ 
tion,  wargaming,  test  and  evaluation,  and  command  and 
control.  He  was  the  first  science  advisor  to  Joint  Forces 
Command  (JFCOM)  and  has  continued  to  support 
JFCOM  and  the  joint  community  in  government  and 
contractor  positions.  He  is  a  government  member  ofITEA 
and  the  JFCOM  staff. 

Endnotes 

1DoD  Testing  in  a  Joint  Environment  Roadmap  signed  by  the  Deputy 
Secretary  of  Defense,  November  12,  2004. 

2Unified  Command  Plan,  Secretary  of  Defense,  October  3,  2008. 

3TSSG.  A  senior  DOT&E  advisory  group  of  flag  officers,  general 
officers,  and  Senior  Executive  Service  members. 

4DoD  Dictionary  of  Military  and  Associated  Terms ,  Joint  Publication  1- 
02,  April  12,  2001. 


5 Doctrine  for  Joint  Operations ,  Joint  Publication  3-0,  September  10, 
2001. 

6DoD  Architecture  Framework  version  2.0,  May  28,  2009. 

7The  JMTAT  WG  was  formed  as  a  sub— working  group  to  the  JSIIL 
concept.  The  JMTAT  WG  represents  a  convergence  of  internal 
USJFCOM  JMT  groups  with  stakeholders  from  other  communities 
across  DoD  including  testing  and  architecture. 

8InterTEC.  An  Office  of  the  Secretary  of  Defense— sponsored  program 
under  the  Central  T&E  Investment  Program  to  develop  and  field  a 
distributed  test  capability  to  conduct  end-to-end  joint  mission  interop¬ 
erability  and  net  ready  testing  in  coordination  with  JMETC. 
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